The actual sins are thus not Hester's adultery, but the minister's cowardice and her former husband's plans of revenge. Society as a whole could not help, but act according to the laws one thought fit to protect it from destruction. The community was blind, but not nearly as guilty of sin as the two men in Hester's life. The narrator reminds the reader of the two most important things a new colony was first raising on its new founded ground: a prison and a cemetery. Death and punishment were the two tools that gave people a certainty and the power...

That is why, although they are guilty of hypocrisy and prejudice, they are having the excuse of being blinded by their struggle to keep their community alive at all costs.
Hester is the element that seemed to threaten the very existence of that community and the only way her fellow citizen knew to react was to outcast her and mark her with a letter that will expose her to public shame every time she would leave the house. In time, she will transform what was supposed to be first a symbol of humiliation into the very sign of advancement toward modernity, open-mindedness and development.